Survey of statisticians on small samples

The Survey of statisticians on small samples was Daniel Kahneman and Amos Tversky’s first direct test of whether sophisticated researchers were good intuitive statisticians. Its decisive finding was unsettling: mathematical expertise did not prevent experts from exaggerating the reliability of small-sample results or from giving poor advice about how much data research required. The survey turned a private confession of error into an empirical demonstration of a widespread cognitive bias.

Causes and Background

Daniel Kahneman had discovered that psychologists often chose samples so small that they faced a 50 percent risk of failing to confirm a true hypothesis. The problem was especially embarrassing because he taught statistics and knew how to calculate safer sample sizes, yet had relied on tradition and intuition when planning his own studies. Strange results in his research, once treated as puzzling findings, now appeared to be artifacts of inadequate sampling.

When Amos Tversky visited the seminar, the two researchers quickly agreed that the optimistic view of intuitive statistical ability was wrong. Their earlier discussion had begun with the question of whether people possessed an intuitive grasp of statistical principles comparable to the intuitive grasp of grammar that a four-year-old displays. The survey sharpened that question: if even researchers selected for mathematical sophistication made systematic mistakes, the difficulty could not be dismissed as ordinary ignorance.

Design and Participants

Kahneman and Tversky constructed a questionnaire around realistic research situations, including the replication of successful experiments. Participants had to select sample sizes, estimate the risk that their studies would fail, and advise hypothetical graduate students planning research. The design therefore tested both abstract statistical understanding and the practical judgments researchers make before collecting evidence.

Tversky administered the questionnaire to sophisticated participants at a meeting of the Society of Mathematical Psychology. The group included the authors of two statistical textbooks, making the sample particularly demanding as a test of expertise. The researchers were not asking whether specialists could perform formal calculations when explicitly prompted; they were examining the intuitive judgments that ordinarily guide decisions about sample size and the credibility of findings.

Findings and Article

The experts greatly exaggerated the likelihood that an original experimental result would be successfully replicated when the sample was small. They also gave very poor advice to the fictitious graduate student about the number of observations she needed to collect. Their responses reproduced the same error Kahneman and Tversky had recognized in themselves: they treated unstable results from small samples as more reliable than sampling variation justified.

The survey thus converted a personal realization into a comparative result. Kahneman was no longer merely one researcher who had trusted defective intuitions; he and Tversky had evidence that the mistake extended to a group chosen for mathematical expertise. Even statisticians, the people most expected to understand sampling variation, were not good intuitive statisticians when they relied on immediate judgment rather than computation.

The findings became the basis of their first joint article, “Belief in the Law of Small Numbers.” Its title was deliberately ironic: human intuitions about random sampling seemed to assume that the law of large numbers applied to small numbers as well. The article urged researchers to regard their statistical intuitions with proper suspicion and to replace impression formation with computation whenever possible. The recommendation was not that researchers abandon judgment altogether, but that they recognize the particular danger of trusting a vivid, coherent impression when the sample is too small to support it.

Method and Consequences

The survey also established the working method that would define the Daniel Kahneman–Amos Tversky collaboration. Their research was conducted as a conversation in which they invented questions, examined their own immediate answers, and treated each question as a small experiment. They were interested in the first intuitive response, including the answer they felt tempted to give even when they knew it was wrong. Shared intuitions became hypotheses about what other people might do, to be tested later in controlled demonstrations.

This method made the survey both self-critical and productive. Kahneman and Tversky did not position themselves as detached experts diagnosing an inferior public; they began by exposing their own deficient intuitions. Their collaboration then used the agreement between them as a clue that the error might be general. The survey’s participants supplied the crucial confirmation, showing that the bias survived training, professional experience, and familiarity with statistical language.

The investigation also anticipated later evidence about prediction and expert judgment. Philip Tetlock’s study of 284 political and economic forecasters gathered more than 80,000 predictions and found that experts performed worse than an equal-probability baseline, even in areas they knew well. Paul Meehl’s review of clinical predictions similarly found that simple statistical rules matched or exceeded trained professionals; across roughly two hundred later comparisons, algorithms were significantly more accurate in about 60 percent of studies, while the remaining comparisons were draws. These findings belong to different research programs, but they extend the survey’s central lesson: expertise does not automatically produce reliable intuitive judgment in uncertain, low-validity environments.

The same lesson appears in calibration studies. Judges who construct subjective probability distributions commonly produce confidence intervals that are too narrow; actual values fall outside the stated extreme bounds in about 30 percent of problems rather than at the much lower rates proper calibration would require. The survey of statisticians on small samples therefore stands within a larger pattern in which experts mistake confidence for accuracy, underrepresent uncertainty, and overlook the corrective force of formulas or larger samples.

The survey’s significance reaches beyond sample-size planning. It helped establish a research program devoted to identifying the intuitive answer before asking whether that answer was statistically sound. In that program, the failure of statisticians was not an anecdote about careless professionals but evidence that human judgment can systematically substitute resemblance, confidence, or coherence for the mathematics of chance. Its enduring consequence was methodological humility: when sampling variation matters, computation is not an optional refinement but a protection against the mind’s persuasive small-number intuitions.